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Summary 



A new study by researchers at the University of California, Irvine, the University of Wisconsin- 
Madison and Policy Studies Associates, Inc. finds that regular participation in high-quality 
afterschool programs is linked to significant gains in standardized test scores and work habits as 
well as reductions in behavior problems among disadvantaged students. These gains help offset 
the negative impact of a lack of supervision after school. The two-year study followed almost 
3,000 low-income, ethnically diverse elementary and middle school students from eight states in 
six major metropolitan centers and six smaller urban and rural locations. About half of the 
young people attended high-quality afterschool programs at their schools or in their 
communities. 

Background on the Study 

The Study of Promising Afterschool Programs was designed to examine relations between high- 
quality afterschool programs and desired academic and behavioral outcomes for low-income 
students. The study was grounded in an assets orientation, which understands that all young 
people, including those living in poverty, have capacities to make healthy, positive choices if 
given the opportunity. The research team reviewed previous research on child and youth 
development in order to depict the processes that lead to positive student outcomes, as shown in 
Exhibit 1. 



Exhibit 1 

Theoretical Linkages between Afterschool Experiences and 
Student Outcomes in the Elementary and Middle Grades 



Dosage 



Personal 
and family 
background 



Child prior 
functioning 



Sets of experiences at: 

• promising after- 
school programs 

• sports, lessons, 
school-based 
activities 

• home supervision 

• unsupervised 
activities 



Intermediate and longer- 
term outcomes, measured 
as: 

• improved social skills 
and interpersonal 
behavior 

• improved grades and 
work habits 

• improved test scores 

• reduced misconduct 
and risky behavior 
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Program Characteristics. The study’s research team identified over 200 candidate programs 
from a review of published materials, recommendations from afterschool experts, and evidence 
from evaluations. Through telephone interviews, document reviews, and site visits, team 
members screened the programs to narrow the list. As a final step, researchers conducted on-site 
interviews and quality-verification observations to confirm the quality of the 35 programs 
selected for the research study. Nineteen programs served elementary school students; 16 
programs served middle school students. Programs were based either in schools or in 
community centers that coordinated with nearby schools. Study sites were geographically 
diverse and included: Aurora, CO; Baldwin, MI; Bridgeport, CT, Central Falls, RI; Denver, CO; 
Los Angeles, CA; Missoula, MT; New York, NY; Oakland, CA; Pawtucket, RI; Salem, OR; San 
Diego, CA; San Ysidro, CA; Seaside, CA. All programs served high concentrations of ethnically 
diverse, low-income youth in high-poverty communities. 

The programs offered services four or five days a week and were free of charge to students. 
Program leaders expected students to participate regularly throughout the school year. Each of 
the selected programs served at least 30 students in one or both of the two age groups studied, 
elementary school children in third or fourth grade and middle school youth in sixth or seventh 
grade. 

The programs had strong partnerships with neighborhoods, schools, and community 
organizations. These partnerships were instrumental in ensuring that the afterschool 
organizations were well established in their communities and were likely to continue operation 
over the two-year study period. 

Because the study was designed to assess the effects of high-quality programs, the research team 
verified each program’s continuing quality during annual visits to conduct interviews and 
observe youth activities. Using a rating system, researchers assessed programs based on 
evidence of supportive relationships between staff and child participants and among participants, 
and on evidence of rich and varied academic support, recreation, arts opportunities, and other 
enrichment activities. Ratings were consistently positive. Students typically were highly 
engaged with one another and with program activities, and group leaders structured activities to 
maximize learning and positive relationships. Adults facilitated activities without imposing 
controls that limited student learning opportunities. Disruptive or chaotic behavior was rarely 
observed; when behavioral disruptions occurred, leaders managed them calmly and 
constructively. 

Through a mix of recreational, arts, and enrichment activities, programs were observed to nurture 
positive interpersonal relationships among students and to actively engage them. Programs 
offered age-appropriate learning opportunities, including tutoring and games designed to 
improve math and reading skills, plus recreational activities, community-based service and other 
experiences, and arts opportunities. Program staff was trained and, in surveys, expressed 
satisfaction with their working environment. Programs maintained low youth-to- staff ratios and 
strong connections with partner schools and with parents. 

Student Characteristics. A total of 2,914 students (1,796 elementary school and 1,118 middle 
school) were studied. At recruitment, the elementary sample was in either third or fourth grade, 
and the middle school sample was in either sixth or seventh grade. The elementary sample was 
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47% male and 89% received free or reduced-price lunch at school; 88% were students of color 
(77% Hispanic, 8% Black, 3% Asian). On average, mothers’ highest educational attainment was 
a high school diploma or GED, and annual family incomes were less than $20,000. 

The middle school sample was 47% male and 63% received free or reduced price school lunch; 
69% were students of color (49% Hispanic, 13% Black, 7% Asian). Mothers had about the same 
level of educational attainment as the elementary group, and average annual family incomes 
were in the $20,000 to $25,000 range. The characteristics of the study participants mirrored the 
characteristics of the schools they attended. 

At the end of the second year, 1,434 of the elementary participants (80% of the recruited sample) 
and 855 of the middle school participants (76% of the recruited sample) remained at the 
participating schools and were available for data collection. 

Participation in Afterschool Programs and Other Activities . Initially the research team sought 
high-quality afterschool programs that operated as stand-alone programs. They soon found, 
however, that many students sampled for the research were participating in multiple afterschool 
experiences in addition to those provided in the sampled programs. Students were also spending 
time supervised at home, and some spent substantial time with no adult supervision at all. 

High-quality afterschool programs were a significant resource for the students, but they 
sometimes competed to attract students who also had access to community centers, sports teams 
or leagues, and churches and other faith-based organizations that hosted recreational programs, 
tutoring, and religious lessons. Also, through provisions of the No Child Left Behind Act, many 
low-performing schools extended the school day with supplementary academic support 
programs. Other options for students at the close of the school day were homes with no adult 
present in the afterschool hours as well as street corners, shopping malls, and other unsupervised 
settings. 

Over the two-year period, 54% of the children in the elementary school sample routinely 
participated in one of the high-quality afterschool programs, typically attending the programs for 
2-3 days a week. Most program children (about two-thirds) did not participate in other activities 
after school and were categorized Program Only. One-third of the program children, however, 
attended the programs for 2-3 days a week while also participating in other activities (organized 
sports, church. Boys and Girls Club, etc.). This group of children was categorized Program 
Plus. About 15% of the elementary school children spent 1-3 days a week unsupervised by 
adults after school, and dropped in sporadically on a mix of sports, school-based activities, and 
academic, arts, or religious lessons. This group was categorized as Low Supervision. 

Almost half (49%) of the middle school sample routinely participated in one of the high-quality 
afterschool programs. Similar to the elementary sample, two-thirds of the program group in 
middle school could be categorized as Program Only. And, one-third of the program group in 
middle school participated in additional activities and were categorized as Program Plus. 

Sixteen percent of the middle school youth were categorized as Low Supervision after school. 

Child Outcome Measures. Classroom teachers and participating youth completed surveys to 
measure the social (social skills with peers, prosocial conduct with peers), academic (grades, task 
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persistence, work habits), and problematic (misconduct, substance use, aggression) functioning 
of study participants. Standardized test scores in reading and math were collected on each child 
through agreements with participating school systems. Data on sampled students were collected 
at three points over a two-year period: baseline, end of Year 1, and end of Year 2. 

Analytic Strategy. Prior to conducting the primary substantive analyses, a multiple imputation 
procedure was used to address missing data due to attrition and failure to complete all 
assessments. In this procedure, missing data are replaced by a sample of observations drawn 
randomly from a multivariate distribution fit to the variable and covariates. The advantage of 
this approach is that all observations are included in the analysis, and missing observations are 
treated as unknown only to the degree that they cannot be reliably inferred from other variables. 
Consequently, the potential for bias in the estimated effects due to missing observations is 
minimized, and the standard errors for model parameter estimates are computed correctly. Ten 
imputed data sets were created in which different samples were selected for missing 
observations, utilizing a Markov chain Monte Carlo procedure implemented using the SAS v9.1 
PROC MI. 

Following imputation of missing data, two-level random-intercept HLM models were fit in 
which students (Level 1) were nested within schools (Level 2) for each child and youth 
developmental outcome. These models allowed researchers to assess change scores in child and 
youth performance across two years with respect to both school factors and individual factors 
including sets or clusters of afterschool experiences. HLM also accounts for the statistical 
dependence that emerges among observations collected in multilevel samples, a common source 
of model misspecification when applying single-level models. 

In the HLM analyses, researchers contrasted changes in scores from baseline to Year 2 for the 
Program Plus vs. Low Supervision groups and Program Only vs. Low Supervision groups. 
These contrasts allowed researchers to examine whether the selected afterschool programs and 
enrichment activities were protective for children and youth who are at risk for social and 
academic problems. Researchers controlled for a number of personal and family characteristics 
that potentially influence participation in various afterschool settings, including child gender and 
ethnicity, and family background (family income, family structure, maternal education, and 
maternal work status). Analyses were conducted separately for the elementary and middle 
school samples. 

In order to evaluate the meaningfulness of findings that were statistically significant, effect sizes 
were calculated and compared to effects from other studies. An effect size is a statistical tool that 
is useful in interpreting the magnitude of the difference between two measures. Unlike a test of 
statistical significance, the effect size is not affected by the size of the samples assembled for the 
study. For readers to understand the relative magnitude of the effect sizes of the findings 
reported below, the following benchmarks based on other recent studies may be useful: 

■ A study of the impact of instruction by Teach For America teachers on math achievement 
found an effect size of 0.15 on math scores after a year of participation in a classroom led by 
a Teach For America teacher (Decker, Mayer, & Glazerman, 2004). 
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■ A study of the impact of the reduction in class size in elementary classrooms by eight 
students per class found an effect size of 0.23 on math scores after one year (Finn & Achilles, 
1999). 

■ In a review of four studies of afterschool programs, Kane (2004) concluded that the expected 
impact of an extra hour of instruction delivered in an afterschool setting over a school year 
equals an effect size of 0.05 in reading and math. 

■ An evaluation of the 21 st Century Community Learning Centers Program in Louisiana found 
that the impact of this afterschool program was an effect size of 0.13 on a combined measure 
of reading, math, and language test scores (Jenner & Jenner, 2007). 

Findings 

Exhibit 2 summarizes the results of the statistical analyses. 

Outcomes of Elementary School Students 

Academic Outcomes 

■ Elementary school students who regularly attended the high-quality afterschool programs 

(alone or in combination with other activities) across two years demonstrated significant 
gains in standardized math test scores, compared to their peers who were routinely 
unsupervised during afterschool hours. Regular participation in the programs was associated 
with gains of 20 percentiles in math achievement test scores over the two-year period for the 
Program Plus group relative to the Low Supervision group (effect size = .73) and 12 
percentiles for the Program Only group relative to the Low Supervision group (effect size = 
•52.) 

■ Program Only and Program Plus students also posted gains in teacher reports of work habits 
(effect sizes of .31 and .35, respectively) and task persistence (.23 and .30, respectively) over 
the two-year period. The students also reported gains in their work habits (effect sizes = .24 
to .41). These gains in work habits and task persistence may have provided important 
support that contributed to the gains in math achievement. 

Social Outcomes 

■ Program Only and Program Plus students posted significant gains in teachers’ reports of 
students’ social skills with peers (effect sizes = .21 to .30) and prosocial behaviors (effect 
sizes = .21 to .23). Program Only and Program Plus students also posted significant 
reductions in aggressive behaviors with peers (effect sizes = .29 to .34). 

Problematic Behaviors 

■ Reductions in elementary students’ reports of misconduct (e.g., skipping school, getting into 
fights) over the two-year period were reported by the Program Only and Program Plus 
groups, relative to unsupervised students (effect sizes of .66 and .51, respectively). 
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Outcomes of Middle School Students 

Academic Outcomes 

■ Middle school students who regularly attended the high-quality afterschool programs (alone 
or in combination with other activities) across two years demonstrated significant gains in 
standardized math test scores, compared to their peers who were routinely unsupervised 
during afterschool hours. Regular participation in the programs was associated with gains of 
12 percentiles in math achievement test scores over the two-year period, relative to students 
who were routinely unsupervised after school. These gains generated effect sizes of .57 for 
the Program Plus group and .55 for the Program Only group, relative to the Low Supervision 
group. 

■ Middle school students who regularly participated in high-quality afterschool programs had 
significant gains in self-reported work habits, relative to unsupervised students (.33 for 
Program Plus and .20 for Program Only). 

Behavioral Outcomes 

■ Reductions in misconduct over the two-year period were reported by Program Plus and 
Program Only middle school students, relative to the Low Supervision group (effect sizes of 
.64 and .55, respectively). 

■ Middle school students who regularly participated in afterschool programs also reported 
reduced use of drugs and alcohol, compared to those in the Low Supervision group. The 
effect sizes (.47 for Program Only and .67 for Program Plus) are four to six times larger than 
those reported in a recent meta-analysis of school-based substance-abuse prevention 
programs aimed at middle school students (Gottfredson & Wilson, 2003). 

Conclusion 

This study found positive outcomes among youth who regularly attended high-quality 
afterschool programs, either alone or in combination with varied sets of additional enrichment 
experiences available in their neighborhoods. In contrast, low supervision coupled with 
intermittent participation in an unstructured program of extra-curricular activities posed 
developmental risks to both elementary school and middle school youth. 

The study focused on economically disadvantaged, minority youth, many of whose fa mi lies were 
recent immigrants. The research team could not know for certain whether the same sets of 
experiences and outcomes would characterize youth in different cultural groups. The findings, 
however, demonstrate the benefits of continuous participation in high-quality afterschool 
programs, community activities, and supervised home settings for youth from economically 
disadvantaged families. 

These findings suggest that plans for high-quality afterschool programming should span entire 
communities. When communities and program providers unite to recruit and engage youth in 
high-quality afterschool experiences, programs can provide the types of benefits described here 
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for the largest number of students. As found in this research, a lack of supervision after school is 
associated with seriously negative outcomes for disadvantaged youth. Working together, youth- 
service providers, schools, local governments, and civic organizations can reach out to youth 
who would otherwise be unsupervised after school and can match them with organized, adult- 
supervised activities in the afterschool hours. 
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Exhibit 2 

Effect Sizes for Statistically Significant Changes from Baseline to Year 2 



Program + Activities vs. 
Low Supervision 


Program Only vs. 
Low Supervision 


Student Outcomes 


Elementary 


Middle 


Elementary 


Middle 


Achievement test scores 










Math 


.73 


.57 


.52 


.55 


Reading 










Student report 










Work habits 


.41 


.33 


.24 


.20 


Misconduct 


-.51 


-.64 


-.66 


-.55 


Substance use 




-.67 




-.47 


Teacher report 










Academic performance 


.25 








Work habits 


.35 




.31 




Task persistence 


.30 




.23 




Social skills with peers 


.30 




.21 




Prosocial with peers 


.21 




.23 




Aggressive with peers 


-.29 




-.34 





Note: An empty cell (e.g., reading achievement test scores) indicates that the finding did not meet the 
study’s baseline criteria for reporting (statistical significance and effect size of .20 or -.20 or larger). 
Shading indicates that the outcome was not measured for that group. 
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